Add vLLM Offline Backend for batch processing

Introduces VLLMOfflineBackend inheriting from VLLMPythonBackend,
using vLLM's synchronous LLM engine for micro-batched inference.
Wires RequestStateStats metrics into response timings. Includes
registration test, docs, and backends.md update.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Fix tokenizer access path for vLLM V1 LLM engine

Use llm.get_tokenizer() instead of llm.llm_engine.tokenizer.tokenizer
which doesn't exist in the V1 engine.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Fix formatting, metrics, and mdformat issues

- Apply ruff format to offline.py
- Fix _wire_vllm_metrics: only use token counts, not monotonic
  timestamps that produce garbage when mixed with wall-clock times
- Apply mdformat to markdown docs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Defer vLLM engine creation to first request and reset CPU affinity

The vLLM LLM engine is now created lazily on the first resolve() call
instead of during process_startup(). This avoids a double-startup that
wastes resources reloading model weights and can cause CPU-affinity
degradation when the engine is created in both the validation path and
the worker process.

Uses double-checked locking with asyncio.Lock to prevent concurrent
engine creation when multiple requests arrive simultaneously.

Adds _reset_cpu_affinity() to restore the full cgroup cpuset before
engine creation, working around OpenMP/torch restricting CPU affinity
in forked worker processes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Wire vLLM RequestStateStats metrics and fix batch lock contention

Map vLLM's monotonic RequestStateStats timestamps to wall-clock
values so TTFT, ITL, and TPOT are reported for offline batches.
Split _process_batch into _take_pending_batch + _run_generate to
release the batch lock during LLM.generate(), and loop the deferred
flush until all pending requests are drained.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Address review: move CPU affinity to common module, use _shutting_down, call shutdown

- Move _reset_cpu_affinity to common.py so both vllm_python backends
  can reuse it; add comment explaining cgroup v2/v1 fallback loop
- Guard resolve() with _shutting_down check to reject requests during
  shutdown
- Call llm.shutdown() before deleting the engine in process_shutdown

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Fix concurrency hazards and clean up batch request dataclass

- Add _generate_lock to serialize LLM.generate() calls, preventing
  overlapping generates on the non-thread-safe sync LLM engine
- Move _shutting_down check + enqueue under the same batch lock
  acquisition so shutdown cannot strand in-flight resolve() waiters
- Set _shutting_down under lock in process_shutdown for atomicity
- Remove unused _BatchedRequest fields (request, request_info,
  request_id) and the uuid import
- Drop duplicate validate_vllm_config; inherited from parent

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Add unit tests for vLLM offline backend

Covers engine laziness, lifecycle/shutdown, batch processing,
deferred flush, generate-lock serialization, shutdown-guard,
and metrics wiring with 31 test cases.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Fix shutdown race, add batch_timeout, and finish test coverage

Shutdown now waits for the deferred-flush task to complete and
acquires _generate_lock before tearing down the engine, preventing
a race where an in-flight LLM.generate() outlives _llm teardown.

Add batch_timeout field (default 0.01s) to VLLMOfflineBackendArgs
so partial batches accumulate for a configurable window before
flushing; full batches still flush immediately.

Add ## WRITTEN BY AI ## docstrings to all 34 test functions, replace
the old cancellation test with test_shutdown_waits_for_inflight_generate,
and add batch_timeout validation tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Replace guide example with throughput profile and remove getattr from metrics

Use throughput,max_concurrency=20 instead of constant,rate=3 in the
offline backend guide example to better reflect the backend's
throughput-oriented purpose.

Replace all getattr calls in _wire_vllm_metrics with hasattr + direct
attribute access per AGENTS.md coding standards.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Reset asyncio locks in process_startup to fix forked worker hang

The benchmark framework calls process_startup/validate/shutdown in
the parent process, then forks worker subprocesses that call
process_startup again. Asyncio locks created on the parent event
loop are unusable in the child, causing the worker to hang and
produce 0 completed requests.

Recreate all asyncio primitives (_batch_lock, _generate_lock,
_engine_lock) and reset batch state in process_startup() so each
worker gets fresh locks bound to its own event loop.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Preload engine in forked workers to fix 0-completion benchmark

The vLLM engine cold-start (42s on CPU) was happening during the
timed benchmark phase inside resolve(), causing SIGTERM to kill the
EngineCore before any requests could complete.

Record the creator PID in __init__ so validate() can detect forked
workers and eagerly preload the engine before the timed phase.
Clear _llm in process_startup so workers never inherit a stale
engine handle from the parent process.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Polish engine lifecycle docs and strengthen lock-reset test

Rewrite the Engine lifecycle section to clearly describe the
three-stage flow: parent preflight (cheap check), worker preload
(PID-based eager engine creation), and resolve() idempotent
fallback.  Mention both fork and spawn workers.

Strengthen test_startup_recreates_locks: use `is not` per lock,
assert batch state is reset, and acquire/release each new lock
to prove they work on the current event loop.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Address PR review comments for vllm_offline backend

- Replace hasattr guards in _wire_vllm_metrics with direct field access
  on vllm.RequestOutput / RequestStateStats; type request_output
  properly and cast at the call site in resolve()
- Add explanatory comment in reset_cpu_affinity() for the cgroup v2/v1
  loop-with-return pattern that reads as odd at first glance
- Call reset_cpu_affinity() in VLLMPythonBackend.process_startup() so
  the regular in-process vLLM backend benefits from the same CPU-affinity
  fix as the offline backend
- Replace 'resolve() fallback' with 'Inference-time safety net' in
  docs to avoid exposing the internal method name to end users

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Fix partial-metrics crash in _wire_vllm_metrics and address review nits

Use getattr with 0.0 defaults for timing fields so that metrics objects
carrying only num_generation_tokens (e.g. test stubs) no longer raise
AttributeError after the metrics-is-None guard. Also update the validate()
docstring to say "inference-time safety net", fix the test_no_metrics_no_crash
docstring, and add a smoke test asserting reset_cpu_affinity() is called in
VLLMPythonBackend.process_startup().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Detect offline worker processes via parent_process() for engine preload

Replace fragile _creator_pid comparison with multiprocessing.parent_process()
so scheduler workers reliably preload the vLLM engine before the timed phase,
avoiding 0-completion runs when cold-start lands inside resolve().

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Fix benchmark progress panel stacking during vLLM offline worker init.

Rich Live's background auto-refresh thread re-rendered the Benchmarks panel
every 250 ms.  Between renders, vLLM worker processes wrote to the inherited
terminal file descriptors during engine initialisation, shifting the cursor
so each re-render landed below the previous one instead of overwriting it.

Switch to manual refresh (auto_refresh=False) and drive repaints explicitly:

* _force_refresh()     — immediate render on state-transition events
                         (benchmark start, complete, finalize)
* _throttled_refresh() — caps update rate at refresh_per_second (~4 fps)
                         during active benchmarking; no-op between events

The window between on_benchmark_start() and the first on_benchmark_update()
now contains zero renders, so worker-process writes cannot cause stacking.
entrypoints.py calls prepare_vllm_benchmark_logging() before workers start
so env-var silencing is inherited by child processes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Suppress vLLM worker startup output and fix offline batch waiter hang.

Worker processes write HF Hub progress bars, tqdm model-loading bars, and
C-level PyTorch warnings (e.g. NUMA) directly to the inherited terminal
file descriptors during engine initialisation.  These writes interleave
with the main-process Rich live panel even when Python-level log levels
are lowered.

Fixes:

* Add suppress_worker_stdio() (utils/terminal.py) — redirects OS-level
  fd 1/fd 2 via dup2() to /dev/null and replaces sys.stdout/sys.stderr,
  silencing both C-level and Python-level writes for the duration.
* Wrap backend.validate() in worker._processing_startup() with
  suppress_worker_stdio() so the vLLM LLM engine cold-start is silent.
* Add VLLM_CONFIGURE_LOGGING=0 to prepare_vllm_benchmark_logging() so
  vLLM's EngineCore subprocesses do not re-run their own log handler
  setup after the level has been lowered.
* Call prepare_vllm_benchmark_logging() in VLLMPythonBackend.process_startup()
  and use vllm_benchmark_engine_config() for the async engine so the
  online backend also benefits from quieter defaults.
* Fix offline batch waiter hang: replace zip(..., strict=True) outside
  the try/except in _run_generate() with an explicit length check that
  routes mismatches through the error path, ensuring all batch waiters
  always receive ready.set() and resolve() calls never hang.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Address low-severity gaps: fd leak, markers, and missing tests.

* terminal.py: store both open(os.devnull) handles before the try block
  and close them in finally, eliminating the two-fd leak per call.
* test_progress.py: add ## WRITTEN BY AI ## module docstring, smoke/sanity
  markers, and descriptive per-test docstrings per AGENTS.md; extract the
  repeated patch target into a module-level constant to stay within the
  79-char line limit.
* test_vllm_offline.py: add test_run_generate_output_count_mismatch_signals
  _all_waiters — verifies the len(outputs) != len(batch) path routes all
  waiters through the error handler so resolve() never hangs.
* test_terminal.py: new POSIX unit test file for suppress_worker_stdio()
  covering stdout/stderr silencing, sys.stdout/stderr restoration,
  fd restoration, fd-leak detection, and exception-safety.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>

Fix offline batch waiter hangs and spawn worker pickling.

Defer asyncio lock creation to process_startup() and strip locks in
pickle state so spawn workers can unpickle the backend. Route all
_run_generate() failures through _signal_batch_failure() so every
batch waiter is unblocked on any exception.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>